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by  GEORGE  N.  BAIRD 

Department  of  the  Navy 
Washington,  D.  C. 


INTRODUCTION 


The  ability  to  benchmark  or  validate  software  to  ensure 
that  design  specifications  are  satisfied  is  an  extremely 
difficult  task.  Test  data,  generally  designed  by  the 
creators  of  said  software,  is  generally  biased  toward  a 
specific  goal  and  tend  not  to  cover  many  of  the  pos- 
sibilities of  combinations  and  interactions.  The  phi- 
losophy of  suggesting  that  “a  programmer  will  never 
do  . . or  “this  particular  situation  will  never  happen” 
is  altogether  absurd.  First,  “never”  is  an  extremely 
long  time  and  secondly,  the  Hagel  theorem  of  pro- 
gramming states  that  “if  it  can  be  done,  whether  absurd 
or  not,  one  or  more  programmers  will  more  than  likely 
try  it.” 

Therefore,  if  a particular  piece  of  software  has  been 
thoroughly  checked  against  all  known  extremes  and  a 
majority  of  all  syntactical  forms,  then  the  Hagel 
theorem  of  programming  will  not  affect  the  software 
in  question.  The  DOD  CCVS  attempts  to  do  just  that 
by  checking  for  the  fringes  of  the  specifications  of 
X3.23-19681  and  known  limits.  It  is  assumed  that  a 
COBOL  compiler  will  perform  satisfactorily  for  the 
audit  routines,  then  it  is  likely  that  the  compiler  sup- 
ports  the  entire  language.  However,  if  the  computer 
viThas  trouble  with  handling  the  routines  in  the  CCVS 
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it  can  be  assumed  that  there  will  indeed  be  other 
errors  of  a more  serious  nature. 

The  following  is  a brief  account  of  the  history  of  the 
DOD  CCVS,  the  automation  of  the  system  and  the 
adaptability  of  the  system  to  given  compilers. 


ACKGROUND 


The  first  revision  to  the  initial  specification  for 
COBOL  (designated  as  COBOlz-liltil2)  was  approved 
by  the  Executive  Committee  of  the  Conference  on 


Data  Systems  Languages*  and  published  in  May  of 
1961.  Recognizing  that  the  language  would  be  subject 
to  additional  development  and  change,  an  attempt 
was  made  to  create  uniformity  and  predictability  in 
the  various  implementations  of  COBOL  compilers. 
The  language  elements  were  placed  in  one  of  two 
categories:  required  and  elective. 

Required  COBOL-1961  consisted  of  language  ele- 
ments (features  and  options)  which  must  be  imple- 
mented by  any  implementor  claiming  a COBOL-1961 
compiler.  This  established  a common  minimum  subset 
of  language  elements  for  COBOL  compilers  and  hope- 
fully a high  degree  of  transferability  of  source  programs 
between  compilers  if  this  subset  was  adhered  to. 

Elective  COBOI/-1961  consisted  of  language  elements 
whose  implementation  had  been  designated  as  op- 
tional. It  was  suggested  that  if  an  implementor  chose 
to  include  any  of  these  features  (either  totally  or 
partially)  he  would  be  expected  to  implement  these 
in  accordance  with  the  specifications  available  in 
COBOL-1961.  This  was  to  provide  a logical  growth 
for  the  language  and  attempt  to  prevent  a language 
element  from  having  contradictory  meaning  between 
the  language  development  specifications  and  im- 
plementor’s definition. 

As  implementors  began  providing  COBOL  compilers 
based  on  the  1961  specifications,  unexpected  problems 
became  somewhat  obvious.  The  first  problem  was  that 
the  specifications  themselves  suggested  mandatory  as 
well  as  optional  language  elements  for  implementing 
COBOL  compilers.  In  addition  the  development  docu- 


• The  Conference  on  Data  System*  Languages  (COPASYL)  is  an 
informal  and  voluntary  organization  of  interested  individuals 
supported  hy  their  institutions  who  contribute  their  efforts  and 
expenses  toward  the  end*  of  designing  and  developing  technique* 
and  languages  to  assist  in  data  systems  analysis,  design,  and 
implementation.  COPASYL  is  responsible  for  the  development 
and  maintenance  of  COBOL. 
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ment  produced  by  CODASYL  was  likely  to  change 
periodically  thus,  providing  multiple  specifications  to 
implement  from.  Compilers  could  consist  of  what  the 
implementor  chose  to  implement  which  would  severely 
handicap  any  chance  of  transferability  of  programs 
among  the  different  compilers,  particularly  since  no  two 
implementors  necessarily  think  alike.  Philosophies  vary 
both  in  the  selection  of  elements  for  a COBOL  compiler 
and  in  the  techniques  of  implementing  the  compiler 
itself.  (As  ridiculous  as  it  may  sound,  some  compilers 
actually  scan,  syntax  check  and  issue  diagnostics  for 
COBOL  words  that  might  appear  in  comments  both 
in  the  REMARKS  paragraph  of  the  Identification 
Division  and  in  NOTE  sentences  in  the  Procedure 
Division.)  The  need  for  a common  base  from  which  to 
implement  became  obvious.  If  the  language  was  to 
provide  a high  degree  of  compatability,  then  all  im- 
plementations had  to  be  based  on  the  same  specifica- 
tion. 

The  second  problem  was  the  reliability  of  the  com- 
piler itself.  If  the  manual  for  the  compiler  indicated 
that  it  supported  the  DIVIDE  statement,  the  user 
assumed  this  was  true.  If  the  compiler  then  accepted 
the  syntax  of  the  DIVIDE  statement,  the  user  as- 
sumed that  the  object  code  necessary  to  perform  the 
operation  was  generated.  When  the  program  executed, 
he  expected  the  results  to  reflect  the  action  represented 
in  his  source  code.  It  appears  that  in  some  cases  perhaps 
no  code  was  generated  for  the  DIVIDE  statement 
and  the  object  program  executed  perfectly  except  for 
the  fact  that  no  division  took  place.  In  another  case, 
when  the  object  program  encountered  the  DIVIDE 
operation,  it  simply  went  into  a loop  or  aborted.  At 
this  point,  the  programmer  could  become  decidedly 
frustrated.  The  source  code  in  his  program  indicated 
that:  (1)  he  requested  that  a divide  take  place,  (2)  there 
was  no  error  loop  in  his  program,  (3)  the  program 
should  not  abort.  This  is  the  problem  we  are  ad- 
dressing: A programmer  should  concern  himself  with 
producing  a source  program  that  is  correct  logically 
and  the  necessary  operating  system  control  statements 
to  invoke  the  COBOL  compiler.  In  doing  so,  he  should 
be  able  to  depend  on  the  compiler  being  capable  of 
contributing  its  talent  in  producing  a correct  object 
program. 

If  the  user  was  assured  that  either:  (1)  each  instruc- 
tion in  the  COBOL  language  had  been  implemented 
correctly,  or,  (2)  that  each  statement  which  was  im- 
plemented did  not  give  extraneous  results,  then  the 
ab'  ve  situation  could  not  exist. 

Thus,  the  need  for  a validation  tool  becomes  ap- 
parent. Although  all  vendors  exercise  some  form  of 
quality  control  on  their  software  before  it  is  released, 


it  is  clear  that  some  problems  may  not  be  detected. 
(The  initial  release  of  the  Navy  COBOL  audit  routines 
revealed  over  50  bugs  in  one  particular  compiler  which 
had  been  released  five  years  earlier.) 

By  providing  the  common  base  from  which  to  imple- 
ment and  a mechanism  for  determining  the  accuracy 
and  correctness  of  a compiler  relative  to  the  specifica- 
tion, the  problem  of  smorgasbord  compilers  (that  may 
or  may  not  produce  expected  results)  should  become 
extinct. 

The  standardization  of  COBOL  began  on  15  January 
1963.  This  was  the  first  meeting  of  the  American  Stan- 
dards Association  Committee,  X3.4.4,*  the  Task  Group 
for  Processor  Documentation  and  COBOL.  The  pro- 
gram of  work  for  X3.4.4  included  . . . “Write  test 
problems  to  test  specific  features  and  combinations  of 
features  of  COB< )L.  Checkout  and  run  the  test  problems 
on  various  COBOL  compilers.”  A working  group 
(X3.4.4.2)  was  established  for  creating  the  “test 
problems”  to  be  used  for  determining  feature  availa- 
bility. 

The  concept  of  a mechanism  for  measuring  the 
compliance  of  a COBOL  compiler  to  the  proposed 
standard  seemed  reasonable  in  view  of  the  fact  that 
other  national  standards  did  indeed  lend  themselves 
to  some  form  of  verifications,  i.e.,  2X4's,  typewriter 
keyboards,  screw  threads. 

IMPLEMENTING  A VALIDATION  SYSTEM 
FOR  COBOL 

In  order  to  implement  a COBOL  program  on  a given 
system,  regardless  of  whether  the  program  is  a valida- 
tion routine  or  an  application  program,  the  following 
must  be  accomplished : 

1.  The  special  characters  used  in  COBOL  (i.e., 

')',  '<'  etc.)  must  be  converted  for  the 

system  being  utilized.! 

2.  All  references  to  implementor-names  within  each 
of  the  source  programs  must  be  resolved. 

3.  Operating  System  Control  Cards  must  be  pro- 


* The  American  Standards  Association  (ASA),  a voluntary 
national  standards  laxly  evolved  to  the  United  States  of  America 
Standards  Institute  (USAS!)  and  finally  the  American  National 
Standards  Institute  (ANSI).  The  committee  X3.4.4  eventually 
became  X3J4  under  a reorganization  of  the  X3  structure.  X3J4  is 
currently  in  the  process  of  producing  a revision  to  X3.23-I968. 
t For  most  computers  the  representatives  for  the  characters 
A-Z,  0-9,  and  the  space  (blank  character)  are  the  same.  However, 
there  is  sometimes  a difference  in  representation  of  the  other 
characters  and  therefore  conversion  of  these  characters  from  one 
computer  to  another  may  be  necessary. 
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duced  which  will  cause  each  of  the  source  programs  to 
be  compiled  and  executed.  Additionally,  the  user  must 
have  the  ability  to  make  changes  to  the  source  pro- 
grams, i.e.,  delete  statements,  replace  statements,  and 
add  statements. 

4.  As  the  programs  are  compiled,  any  statements 
that  are  not  syntactically  acceptable  to  the  compiler 
must  be  modified  or  “deleted”  so  that  a clean  compila- 
tion takes  place  and  an  executable  object  program  is 
produced. 

5.  The  programs  are  then  executed.  All  execution 
time  aborts  must  be  resolved  by  determining  what 
caused  the  abort  and  after  deleting  or  modifying  that 
particular  test  or  COBOL  element,  repeating  steps  3 
and  4 until  a normal  end  of  job  situation  exists. 

Development  of  audit  routines 

March  I9fi3,  X3.4.4.2  (the  Compiler  Feature  Availa- 
bility Working  Group)  began  its  effort  to  create  the 
COBOL  programs  which  would  be  used  to  determine 
the  degree  of  conformance  of  a compiler  to  the  proposed 
standard.  The  intent  of  the  committee  was  not  to  fur- 
nish a means  for  debugging  compilers,  but  rather  to 
determine  “feature  availability.”  Feature  availability 
was  understood  to  mean  that  the  compiler  accepted  the 
syntax  and  produced  object  code  to  produce  the  de- 
sired result.  All  combinations  of  features  were  not  to 
be  tested;  only  a carefully  selected  sample  of  features 
(singly  and  in  combination)  were  to  be  tested  to  insure 
that  they  were  operational.  The  test  programs  them- 
selves were  to  produce  a printed  report  that  would 
reflect  the  test  number  and  when  possible  whether  the 
test  “Passed”  or  “Failed.”  See  Figure  1. 

When  a failure  was  detected  on  the  report,  the  user 
could  trace  the  failure  to  the  source  code  and  attempt 

Source  Statement* 

TEST-0001. 

MOVE  001  TO  TEST-NO. 

MOVE  ZERO  TO  ALPHA. 

ADD  1 TO  ALPHA. 

IF  ALPHA  - 1 PERFORM  PASS  ELSE  PERFORM  FAIL. 

TEST-0002. 

Result* 

TEST  NO  P - F 

ADD  1 P 


ADD  21  F 

Figure  1 Example  of  X.'1.4.4.2  teat  and  printed  result* 


to  identify  the  problem.  The  supporting  code  (printing 
routine,  pass  routine,  fail  routine,  etc.)  was  to  be  written 
using  the  most  elementary  statements  in  the  low-level 
of  COBOL.  The  reason  for  this  was  twofold . 

1.  The  programs  would  be  able  to  perform  on  a 
• minimum  COBOL  compiler  (Nucleus  level  1, 

Table  Handling  level  1,  and  Sequential  Access 
level  1). 

2.  The  chances  of  the  supporting  code  not  being 
acceptable  to  the  compiler  being  tested  were 
lessened. 

The  programs,  when  ready,  would  be  provided  in 
card  deck  form  along  with  the  necessary  documenta- 
tion for  running  them.  (The  basic  philosophies  of 
design  set  forth  by  X3.4.4.2  were  carried  through  all 
subsequent  attempts  to  create  compiler  validat'on 
systems  for  COBOL.) 

Assignments  were  made  to  the  members  of  the  com- 
mittee and  the  work  began.  This  type  of  effort  at  the 
committee  level,  however,  was  not  as  productive  as 
the  work  of  standardizing  the  language  itself. 

In  April  1967,  the  Air  Force  issued  a contract  for  a 
system  to  be  designed  and  implemented  which  could 
be  used  in  measuring  a compiler  against  the  standard. 
The  Air  Force  COBOL  Compiler  Validation  System 
was  to  create  test  programs  and  adapt  them  to  a given 
system  automatically  by  means  of  fifty-two  parameter 
cards. 

The  Navy  COBOL  audit  routines 

In  August  of  1967,  The  Special  Assistant  to  the 
Secretary  of  the  Navy  created  a task  group  to  influence 
the  use  of  COBOL  throughout  the  Navy.  Being  aware 
of  both  the  X3.4.4.2  and  Air  Force  efforts,  (as  well 
as  the  time  involved  for  completion),  a short  term 
project  was  established  to  determine  the  feasibility' 
of  validating  COBOL  compilers.  After  examining  the 
information  and  test  programs  available  at  that  time, 
the  first  set  of  routines  was  produced.  In  addition  to  the 
original  X3.4.4.2  philosophy,  the  Navy  added  the 
capability  of  providing  the  result  created  by  the  com- 
puter as  well  as  the  expected  result  when  a test  failed. 
Also,  instead  of  a test  number,  the.  actual  procedure 
name  in  the  source  program  was  reflected  in  the  output. 
See  Figure  2. 

The  preliminary  version  of  the  Navy  COBOL  audit 
routines  was  made  up  of  12  programs  consisting  of 
about  .1000  lines  of  source  code.  The  tailoring  of  the 
programs  to  a particular  compiler  was  done  by  hand 
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(by  physically  changing  cards  in  the  deck  or  by  using 
the  vendor's  software  for  updating  COBOL  programs). 
As  tests  were  deleted  or  modified,  it  was  difficult  to 
bring  the  programs  back  to  their  virgin  state  for  sub- 
sequent runs  against  different  compilers  or  for  de- 
termining what  changes  had  to  be  made  in  order  that 
the  programs  would  execute. 

This  was  a crude  effort,  but  it  established  the  neces- 
sary evidence  that  the  project  was  feasible  to  continue 
and  defined  techniques  for  developing  auditing  systems. 
Because  of  the  favorable  comments  received  on  this 
initial  work  done  by  the  .Vavy,  it  appeared  in  the  best 
interest  of  all  to  continue  the  effort. 

After  steady  development  and  testing  for  a year, 
Version  4 of  the  Navy  COBOL  Audit  Routines  was 
released  in  December  1969.  The  routines  consisted  of 
55  Programs,  consisting  of  18,000  card  images  capable 
of  testing  the  full  standard.  The  routines  had  also  be- 
come one  of  the  benchmarks  for  all  systems  procured 
by  the  Department  of  the  Navy  in  order  to  ensure  that 
the  compiler  delivered  with  the  system  supported  the 
required  level  of  American  National  Standard  COBOL.* 

Also,  Version  4 introduced  the  VP-Routine,  a pro- 
gram that  automated  the  audit  routines.  Based  on 
fifty  parameter  cards,  all  implementor-names  could 
be  resolved  and  the  test  programs  generated  in  a one- 
pass  operation.  See  Figure  3. 

In  addition,  by  coding  specific  control  cards  in  the 
Working-Storage  Section  of  the  VP-Routine  as  con- 
stants, the  output  of  the  VP-Routine  became  a file 
that  very  much  resembled  the  input  from  a card  reader, 
i.e. , control  cards,  programs,  etc. 

Bv  specifying  the  required  Department  of  Defense 
COBOL  subset  of  the  audit  routines  to  be  used  in  a 
validation,  only  the  programs  necessary  for  validating 

Source  Statements 

ADD-TEST- 1. 

MOVE  I TO  ALPHA. 

ADI)  1 It)  ALPHA 

IF  ALPHA  =2  PERFORM  PASS  ELSE  PERFORM  FAIL. 
Reunite 

FEATURE  PARAGRAPH  P/F  COMPUTED  EXPECTED 

ADD  ADD-TEST-1  FAIL  1 2 

ADD  ADD-TEST-2  PASS 

Figure  2—  Example  of  Navy  test  and  printed  results 

* ■ ;i  llHiH,  the  Department  of  Defense,  realizing  that  several 
thousand  combinations  of  modulex/levels  were  possible,  estab- 
lished four  subsets  of  American  National  Standard  COBOL  for 
procurement  purposes. 


V-P  Routine  Input: 

X-0  SOURCE-COMPUTER-NAME 
X-l  OBJECT-COMPUTER-NAME 
X-3 


X-8  PRINTER 

X-9  CARD-READER 

X-10 

X-50 

Audit  Routine  File: 

SOURCE-COMPUTER. 

XXXXXO 

SELECT  PRINT-FILE  ASSIGN  TO 
XXXXX8 

The  audit  routine  after  processing  would  be: 

SOURCE-CO  M PUTER. 

SOURCE-COMPUTER-NAME. 


SELECT  PRINT-FILE  ASSIGN  TO 
PRINTER 

Figure  3 — Example  of  input  to  the  support  routine,  Population 
file  where  audit  routines  are  stored  and  resolved  audit  routine 
after  processing 

that  subset  of  elements  or  modules  would  be  selected, 
i.e.,  SUBSKT-A,  B,  C,  or  D.  The  capability  also  existed 
to  update  the  programs  as  the  “card  reader"  file  was 
being  created.  The  use  of  the  VP-Routine  was  not 
mandatory  at  this  time,  but  merely  to  assist  the  person 
validating  the  compiler  in  setting  up  the  programs  for 
compilation.  Once  the  VP-Routine  was  set  up  for  a 
given  system,  there  was  little  trouble  running  the  audit 
routines.  The  user  then  had  only  to  concern  himself 
with  the  validation  itself  and  with  achieving  successful 
results  from  execution  of  the  audit  routines.  When  an 
updated  set  of  routines  was  distributed,  there  was  no 
effort  involved  in  replacing  the  old  input  tape  to  the 
VP-Routine  with  the  new  tape. 

The  Air  Force  COBOL  audit  routines 

The  Air  Force  COBOL  Compiler  Validation  System 
(AFCCVS)  was  not  a series  of  COBOL  programs  but 
rather  a test  program  generator.  The  user  could  select 
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Source  statement  in  test  library 

T 1N078A101NUC,  2NUC  4U 

400151  77  WRK-DS-18V00  PICTURE  89(18). 

400461  77  A 180NES-DS-  18VOO  PICTURE  89(18). 


444047  ] 

400881  77  A180NES-CS-18VOO 
400891 

802925  TEST-1  NUC-078. 

802930  MOVE  A180NES-DS-18VOO 

802935  ADD  A180NES-CS-I8VOO 

802940  MOVE  WRK-DS-18VOO 

802945  MOVE  ‘222222222222222222’ 

802950  MOVE  ‘1N078’ 

802955  PERFORM  SUPPORT-RTN 

Test  results 


VALUE  11111111111111111. 

PICTURE  S9(  18)  COMPUTATIONAL 
VALUE  llllllllllllllllll. 

TO  WRK-DS-18VOO. 
TO  WRK-DS 18VOO 
TO  SUP-WK-A. 

TO  SUP-WK-C. 

TO  SUP-ID- WK-A 

HRU  SUP-TRN-C. 


.1N078  .1N079. 

.222222222222222222.09900. 


Figure  4 — Example  of  Air  Force  test  and  printed  results 


the  specific  tests  or  modules  he  was  interested  in  and 
the  AFCCVS  would  create  one  or  more*  programs  from 
a file  of  specific  tests  which  were  then  compiled  as  audit 
routines.  Implementor-names  were  resolved  as  the 
programs  were  generated  based  on  parameter  cards 
stored  on  the  test  file  or  provided  by  the  user. 

The  process  required  several  passes,  including  the 
sorting  of  all  of  the  selected  tests  to  force  the  Data 
Division  entries  into  the  Data  Division  and  place 
the  tests  themselves  in  the  Procedure  Division  where 
they  logically  belonged.  An  additional  pass  was  re- 
quired to  eliminate  duplicate  Data  Division  entries 
(more  than  one  test  might  use  the  same  data-item  and 
therefore  there  would  be  more  than  one  copy  in  the 
Data  Division).  See  Figure  4. 

Still  another  program  was  used  to  make  changes  to 
the  source  programs  as  the  compiler  was  validated. 
As  in  the  Navy  system,  certain  elements  had  to  be 
eliminated  because:  (1)  they  were  not  syntactically 
acceptable  to  the  compiler  or,  (2)  they  caused  run  time 
ab<  >rts. 

Deportment  of  Defense  COBOL  validation  system 

In  December  1970,  The  Deputy  Comptroller  of  ADP 
in  the  Office  of  the  Secretary  of  Defense  asked  the 
Navy  to  create  what  is  now  the  DOD  Compiler  Valida- 
tion System  for  COBOL  taking  advantage  of:  (1)  the 
better  features  of  both  the  Navy  COBOL  Audit  Rou- 
tines (Version  4)  and  the  Air  Force  CCVS  and  (2)  the 
four  years  of  in-house  experience  in  designing  and  im- 
plementing audit  routines  on  various  systems  as  well  as 
the  actual  validation  of  compilers  for  procurement 
purposes. 


The  Compiler  Validation  System  (of  which  the  sup- 
port program  was  written  in  COBOL)  had  to  be  readily 
adaptable  to  any  computer  system  which  supported  a 
COBOL  compiler  and  which  was  likely  to  be  bid  on  any 
RFP  issued  by  the  Department  of  Defense  or  any  of 
its  agencies.  It  also  had  to  be  able  to  communicate  with 
the  operating  system  of  the  computer  in  order  to  pro- 
vide an  automated  approach  to  validating  the  COBOL 
compiler.  The  problem  of  interfacing  with  an  operating 
system  may  or  may  not  be  readily  apparent  depending 
on  whether  an  individual  is  more  familiar  with  IBM’s 
Full  Operation  System  (OS),  which  is  probably  the 
most  complex  operating  system  insofar  as  establishing 
communication  between  itself  and  the  user  is  con- 
cerned, or  with  the  Burroughs  Mjister  Control  Program 
(MCP),  where  the  control  language  can  be  learned  in  a 
fifteen  or  twenty  minute  discussion. 

Since  validating  a compiler  may  not  be  necessary 
very  often,  the  amount  of  expertise  necessary  for  com- 
municating with  the  CVS  should  be  kept  to  a minimum. 
The  output  of  the  routines  should  be  as  clear  as  possible 
in  order  not  to  confuse  the  reviewer  of  the  results  or  to 
suggest  ambiguities. 

The  decision  was  made  to  adopt  the  Navy  support 
system  and  presentation  format,  for  several  reasons. 

(1)  It  would  be  easier  to  introduce  the  Air  Force  tests 
into  the  Navy  routines  as  additional  tests  because  the 
Navy  routines  were  already  in  COBOL  program  format . 
It  would  have  been  difficult  to  recode  each  of  the  Navy 
tests  into  the  format  of  specific  tests  on  the  Air  Force 
Population  File  because  of  the  greater  volume  of  tests, 

(2)  The  Navy  support  program  had  become  rather 
versatile  in  handling  control  cards,  even  for  IBM’s 
08,  whereas  the  Air  Force  system  had  only  limited 
control  card  generation  capability. 
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The  merging  of  the  Air  1 tree  and  Nary  routines 


Source  statement* 


The  actual  merging  of  the  routines  started  in 
February  1971  and  continued  until  September  1971. 
During  the  merging  operation,  it  was  noted  that  there 
was  very  little  overlap  or  redundancy  in  the  functions 
tested  by  the  Air  Force  and  Navy  systems.  In  actuality, 
the  two  sets  of  tests  complemented  each  other.  This 
could  only  be  attributed  to  the  different  philosophies 
of  the  two  organizations  which  originally  created  tin* 
routines.  For  example  in  the  tests  for  the*  ADD  state- 
ment : 

A ir  Force  Nary 

signed  fields  unsighed  fields 

most  fields  18  digits  long  most  fields  1-10  digits 

long 

more  computational  more  display  items 

items 

After  examining  the  Add  tests  for  the  combined  DOD 
routines,  it  was  noticed  that  a few  areas  had  been 
totally  overlooked. 


ADD-TEST  t 
MOVE  1 TO  ALPHA 

ADD  1 TO  ALPHA 
IK  ALPHA  m 2 

PERFORM  PASS 


Initialization  if 
necessary. 

The  Test 

Check  the  results  fie 
test  and  hand.  ■ the 
accounting  of  that 


ELSE 

GO  T<  i ADD-1 'A  ■ ii-l. 
GO  TO  ADD-WRI'J  j . I 


ADD  -DELETK-1 
PERFORM  DELETE. 
GO  TO  ADD-WHITE-1 


Norma!  exit  path  to  ti 
write  paragraph 
Aim-irma]  path  to  'lie 
write  statement  if  tin- 
iest is  deleted  via  the 


NOTE  statement. 
Correct  and  computed 
results  are  formatted 
(or  printing 


ADD-FA1L-1. 

MOVE  ALPHA  TO  ('OMITTED 
MOVE  '2'  TO  CORRECT 
PERFORM  FAIL 
ADD-WRITE-! 

MOVE  ADD-TEST  1 TO  PARAGRAPH  NAME. 
PERFORM  PI  NT  RESl  L I S 
ADD-TEST-2. 


Results  are  printed. 


Figure  5-  Example  of  DOD  test  and  supporting  code 


1.  An  ADD  statement  that  forced  the  “temp” 
used  by  the  compiler  to  hold  a number  greater 
than  18  digits  in  length : 

i.e.,  ADD  +999999999999999999 
+ 999999999999999999 
+999999999999999999 
-999999999999999999 
-999999999999999999 
-99  TO  ALPHA 

. . . where  the  intermediate  result  would  be 
greater  than  18  digits,  but  the  final  result  would 
be  able  to  fit  in  the  receiving  field. 

2.  There  were  not  more  than  eight  operands  in 
any  one  ADD  test. 

3.  A size  error  tost  using  a COMPUTATIONAL 
field  when  the  actual  value  could  be  greater 
than  the  described  size  of  the  field,  i.e.,  ALPHA 
PICTURK  9(4)  COMP.  . . specifies  a data  item 
that  could  contain  a maximum  value  of  9999 
without  tin  overflow  condition;  however,  because 
the  field  may  be  set  up  internally  in  binary,  the 
decimal  value  may  be  less  than  the  maximum 
binary  value  it  could  hold: 

.Maximum  COBOL  value  = 9999 
Maximum  hardware  value~16383 

Therefore,  from  this  point  of  view,  the  merging  of 


the  routines  disclosed  the  holes  in  the  validation  sys- 
tems being  used  prior  to  t In*  current  DOL)  routines. 

The  general  format  of  each  test  is  made  up  of  several 
paragraphs:  (1)  the  actual  “test''  paragraph;  (2)  a 
“delete”  paragraph  which  takes  advantage  of  the 
COBOL  NOTH  for  deleting  tests  which  the  compiler 
being  validated  cannot  handle;  (3)  the  fail”  paragraph 
for  putting  out  the  computed  and  correct  results  when 
a test  fails;  and  t4)  a "write”  paragraph  which  places 
the  test  name  in  the  output  line  and  causes  it  to  be 
written.  See  Figure  5. 

The  magnitude  of  the  size  of  the  D<  )D  Audit  Routines 
was  approaching  HHl.(KK)  lines  of  source  coding, 
making  up  130  programs.  The  number  of  environ- 
mental changes  (resolution  of  implementor-names)  was 
in  the  neighborhood  of  1,00(1  and  thenumhi  r of  operat- 
ing system  control  cards  required  to  execute  the 
program  would  be  from  1 .300  to  r>.0*H » depending  on  the 
complexity  of  the  operating  system  involved. 

This  was  where  the  support  program  could  save  a 
large  amount  of  both  w..rk  and  mistakes  The  Versatile 
Program  Management  System  (VP. MSI)  was  designed 
to  handle  all  of  these  problems  with  a minimum  of 
effort. 

Versatile  program  management  system  (Vl'MSl) 

A good  portion  of  the  merging  included  additional 
enhancements  to  the  VP.MS1  (support  program) 
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which,  by  this  time,  through  an  evolutionary  process 
had  learned  to  manage  two  new  languages;  FORTRAN 
and  JOVIAL.  The  program  had  been  modified  based 
on  the  additional  requirements  of  various  operating 
systems  for  handling  particular  COBOL  problems; 
the  need  for  making  the  system  easy  for  the  user  to 
interface  with,  and  the  need  to  provide  all  interfaces 
between  the  user,  the  audit  routines,  and  the  operating 
system. 

The  introduction  of  implementor  names  through  “X-cards" 

The  first  problem  was  the  resolution  of  implementor- 
names  within  the  source  COBOL  programs  making  up 
the  audit  routines.  In  the  COBOL  language,  particularly 
in  the  Environment  Division,  there  are  constructs  which 
must  contain  an  implementor-defined  word  in  order  for 
the  statement  to  be  syntactically  complete.  Figure  6 
shows  where  the  implementor-names  must  be  provided. 

THE  NOTE  placed  as  the  first  word  in  the  para- 
graph causes  the  entire  paragraph  to  be  treated  as 
comments.  Instead  of  the  “GO  TO  ADD-WRITE-1” 
statement  being  executed,  the  logic  of  the  program  falls 
into  the  delete  paragraph  which  causes  the  output  re- 
sults to  reflect  the  fact  that  the  test  was  deleted. 

If  the  syntax  error  is  in  the  Data  Division,  then  the 
coding  itself  must  be  modified.  VPMSl  shows,  in  its 
own  printed  output,  the  old  card  image  as  well  as  the 
new  card  image  so  that  what  has  been  altered  is  readily 
apparent,  i.e., 

012900  02  A PIC  ZZ9  Value  T.  NC1085.2  OLD 
012900  02  A PIC  ZZ9  Value  1.  NC108*RE  NEW 

ENVIRONMENT  DIVISION. 

SOURCE-COMPUTER. 

implementor-name- 1. 

OBJECT-COMPUTER. 

implementor-name-2. 

SPECIAL-NAMES. 

implementor-name-3  is  MNEMONIC-NAME 


FILE-CONTROL 

SELECT  FILE  NAME  ASSIGN  TO  implementor-name-4. 


data  division 

FD  FILE  NAME 

VALUE  OF  implementor-name-5  IS  impleinentor-defined. 

Figure  6-  Implementor  defined  names  that  would  ap|>ear 
in  a COBOL  program 


If,  while  executing  the  object  program  of  an  audit 
routine,  an  abnormal  termination  occurs,  then  a change 
is  required.  The  cause  might  be,  for  example,  a data 
exception  or  a program  loop  due  to  the  incorrect  im- 
plementation of  a COBOL  statement.  In  any  case,  the 
test  in  question  would  have  to  be  deleted.  The  NOTE 
would  be  used  as  specified  above. 

In  addition,  VPMSl  provides  a universal  method 
of  updating  source  programs  so  that  the  individual  who 
validates  more  than  one  compiler  is  not  constantly  re- 
quired to  learn  new  implementor  techniques  for  up- 
dating source  programs. 

Example  of  update  cards  through  VPMSl: 

012900  02  A PIC  ZZ9  (If  the  sequence  number  is 
VALUE  1.  equal  the  card  is  replaced ; 

01321C  MOVE  1 TO  A.  if  there  is  no  match  the 
014310  NOTE  card  is  inserted  in  the  ap- 

propriate place  in  the 
program.) 

014900*  (Deletes  card  014900) 

029300*099000  (Deletes  the  series  from  0293(H) 
through  099000). 

To  carry  the  problem  a step  further.  Some  of  the 
names  used  by  different  implementors  for  the  high 
speed  printer  in  the  SELECT  statement  have 
been  PRINTER,  SYSTEM-PRINTER,  FORM- 
PRINTER,  SYSOUT,  SYSOU1,  PI  FOR  LIST- 
ING, ETC.  It  is  obvious  to  a programmer  what,  the 
implementor  has  in  mind,  but  the  compiler  that  expects 
SYSTEM-PRINTER,  will  certainly  reject  any  of 
the  other  names.  Therefore,  each  occurrence  of  an 
implementor-name  must  be  converted  to  the  correct 
name.  The  approach  taken  is  that  each  implementor- 
name  is  defined  to  VPMSl.  For  example,  the  printer 
is  known  as  XXXX36  and  the  audit  routines  using 
the  printer  would  be  set  up  in  the  following  way: 

SELECT  PRINT-FILE  ASSIGN  TO 
XXXXX36 

And  the  user  would  provide  the  name  to  be  used  by  the 
computer  being  tested  through  an  “X-OARD." 

X-36  SYSTEM-PRINTER 
VPMSl  would  then  replace  all  references  of  XXXXX3(> 
with  SYSTEM-PRINTER. 

SELECT  PRINT-FILE  ASSIGN  TO 

SYSTEM-PRINTER. 

Ability  to  update  programs 

The  next  problem  was  to  provide  the  user  with  a 
method  for  making  changes  to  the  audit  routines  in 
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ADI  > TEST  1 

N't !t  sorted  by  the  user  as  an  update  to  the  program.) 
MOVE  1 TO  ALPHA. 

TO  TO  A L)D- WRITE-1. 

ADD-DELETE- 1 

perform  delete. 


Figure  7— Example  of  deleting  a test  in  the  DOD  CCVS 

an  (.rdi't  iv  fashion  and  at  the  same  time  provide  a maxi- 
mum amount  of  documentation  for  each  change  made. 
Thetv  are  two  reasons  for  the  user  to  need  to  make 
mudiiications  to  the  actual  audit  routines: 

a.  If  'he  compiler  will  not  accept  a form,  of  syntax 
it  must  be  eliminated  in  order  to  create  a syn- 
ta  ally  correct  program.  There  are  two  ways 
to  accomplish  this.  In  the  Procedure  Division 
the  NOTE  statement  is  used  to  force  the  “in- 
i’ al id ' ' statements  to  become  comments.  The 
resu’  . of  this  action  would  cause  the  test  to  be 
deleted  and  this  would  be  reflected  in  the  out- 
put. See  Figure  7. 

i 'RELATING  SYSTEM  CONTROL  CARD 
GENERATION 

Tin  third  problem  was  the  generation  of  operating 
system  control  cards  in  the  appropriate  position  relative 
to  da  source  programs  in  order  for  the  programs  to  be 
compiled,  loaded  and  executed.  This  was  the  biggest 
ehallcngi  for  YPMS1;  a COBOL  program  which  had 
to  be  structurally  compatible  with  all  COBOL  com- 
piler-. and  which  also  had  to  be  able  to  interface  with 
.11  operating  systems  with  a negligible  amount  of 
modification  for  each  system. 

The  philosophy  of  the  output  of  VP.MSl  is  a file 
acceptable  to  a particular  operating  system  as  input. 
For  the  most  part  this  file  closely  resembles  what  would 
1 r'  l e introduced  to  the  operating  system  through 
in's  input  device  or  card  reader,  i.e.,  control 
ds  source  program,  data,  etc. 

' ■ e veneration  of  operating  system  control  cards  is 
■ I,  the  specific  placement  of  the  statement  and 
M requirement  or  need  for  specific  statements  to  ac- 
mpli.-h  additional  functions.  These  control  cards  arc 
re>  at  d to  VP.MSl  in  a form  that  will  not  be  inter- 
" V * 1 by  tl  operating  system  and  arc  annotated  as 


to  their  appropriate  characteristic  body  t1 

actual  control  card  starts  in  position  8 of  the  in: 
record.  Position  one  is  reserved  for  a code  that  specifies 
the  type  of  control  card  The  following  is  allowed  in 
specifying  control  cards:  Initial  control  cards  are 
generated  once  at  the  beginning  of  the  file.  Beginning 
control  cards  are  generated  before  each  source  program 
with  a provision  for  specifying  control  cards  which 
are  generated  at  specific  times,  i.e.,  JOB  type  cards 
subroutine  type  cards,  library  control  cards,  etc.  End- 
ing control  cards  are  generated  after  each  source  pro- 
gram with  the  same  provision  as  beginning  control 
cards.  Terminal  control  cards  are  generated  prior  to 
the  file  being  closed.  Additional  control  cards  are 
generated  for  assigning  hardware  devices  to  the  object 
program,  bracketing  data  and  for  assigning  work  areas 
to  be  used  by  the  COBOL  Sort. 

Th  re  are  approximately  25  files  used  by  the  entire 
set  of  validation  routines  for  which  control  cards  may 
need  to  be  prepared.  In  addition  to  the  control  cards 
and  information  for  the  Environment  Division,  the 
total  number  of  control  statements  printed  for  VP.MSl 
could  bo  in  the  neighborhood  of  200  card  images  and 
the  possible  number  of  generated  control  cards  on  the 
output  file  could  be  as  large  as  5000.  The  saving  in  time 
and  JCL  errors  that  could  be  prevented  should  be 
obvious  at  this  point. 

This  Environmental  information  need  not  be  pro- 
vided by  the  user  because  once  a set  of  VP.MSl  control 
cards  has  been  satisfactorily  debugged  on  the  system 
in  question,  they  can  be  placed  in  the  library  file  that 
contains  the  same  program  so  that  a single  request 
could  extract  the  VP.MSl  control  cards  for  a given 
system. 


CONCLUSION 

It  has  been  demonstrated  that  the  validation  of  COBOL 
compilers  is  possible  and  that  the  end  result  is  bene- 
ficial to  both  compiler  writers  and  the  users  of  these 
compilers.  The  case  with  which  the  DOD  CCVS  can 
be  automatically  adapted  to  a given  computer  system 
has  eliminated  approximately  85  to  90  percent  of  the 
work  involved  in  validating  a COBOL  compiler. 

Although  most  compilers  are  written  from  the  same 
basic  specifications  (i.e.,  the  American  National  Stan- 
dard COBOL,  X3. 23-1908,  or  the  CODASYL  COBOL 
Journal  of  Development)  the  results  are  not  always 
the  same.  The  DOD  CCVS  1ms  exposed  numerous 
compiler  hugs  as  well  as  misinterpretations  of  the 
language.  Due  to  this  and  similar  efforts  in  the  area  of 


' 


The  DOD  COBOL  Compiler  Validation  System 


r 

all  contributing  to  better  software,  both  at  the  compiler 
and  the  application  level. 
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compiler  validation,  the  compatibility  of  today’s 
compilers  has  grown  to  a high  degree. 

We  are  now  awaiting  the  next  version  of  the  American 
National  Standard  COBOL.  The  new  specifications 
will  provide  an  increased  level  of  compatibility  between 
compilers  because  the  specifications  are  more  definitive 
and  contain  fewer  “implementor  defined”  areas.  In 
addition,  numerous  enhancements  and  several  clari- 
fications have  been  included  in  the  new  specification — 
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